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ABSTRACT 



This paper describes and analyzes the models and methods 
used to evaluate asynchronous online learning networks and presents a 
"Systems Component Contexture Model" for conceptualizing the evaluation of 
online learning networks. The evaluation team finds it useful to divide the 
variables they examine into descriptive, process, and outcome variables. The 
complexity of the relationships of these variables has resulted in the 
development of a model based on systems theory and research that can be used 
as a unifying conceptual tool for component analysis and further research. 

The model consists of examining the influence of four components (Content, 
Context, Collaboration, and Control) of the online learning network at three 
levels: institutional experience, all aspects of the mediated curriculum, and 
the learner level. The use of this holistic model provides a framework from 
which to evaluate and to examine the questions left unasked, even as it 
provides the opportunity to draft new questions for the future. (Contains 1 
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Pnarpose 

The development of appropriate methodologies for evaluating the myriad, ever- 
changing forms of online learning presents a critical challenge to distance educators. 

The open-ended nature of online learning projects, the multiple threads of conversation, 
and fluid participation patterns call for new ways of looking at evaluation. The purpose 
of this paper is to critically analyze the models and methods we have used to evaluate 
asynchronous online learning networks, and to present a “Systems Component 
Contexture Model” that we have developed for conceptualizing the evaluation of online 
learning networks. We have selected three questions for discussion. For each question, 
we will discuss the models and methods used to examine the question, the theoretical 
framework or perspective underlying each model, mode of inquiry, data sources, and a 
critical analysis of the application of the model. 



We have previously defined evaluation as a 

"... systematic and purposive inquiry that includes the collection, analysis, and 
reporting of data relating to the efficiency, appropriateness, effectiveness, and value 
of operational characteristics and outcomes of a procedure, program, process, or 
product. ” 

Traditional evaluation methods used to evaluate the effectiveness of learning within 
the four walls of a classroom do not transfer well to the varied contexts included within the 
umbrella designation of online learning networks. Because a “network” may be used in 
distinguishing any set of people, organizations, or other social entities connected by socially 
meaningful relationships the term can be broadly applied to many learning projects from 
course-based instruction to professional development conferences. In addition, online 
learning designs are often based on constructivist, learner-centered principles which provide 
more learner control, facilitate the sharing of multiple perspectives, and place emphasis on 
the collaborative construction of knowledge occurring within a group of learners. 

Objectives, either behavioral or cognitive, with a stated outcome for all learners within a 
fixed period of study are not the goal of many online learning projects. We, as evaluators are 
challenged to understand the unique characteristics of the sociotechnical network created in 
the online medium and its social, organizational, and ecological structure in order to develop 
new principles for examining and evaluating learning. 

Our evaluation team finds it useful to divide the variables we examine into three 
categories: descriptive, process, and outcome variables. Descriptive variables address the 
context, setting, and characteristics of the participants before the course, program or learning 
event. Included among descriptive variables we have studied are the organizational setting, 
demographic characteristics of faculty, staff and learners, and the participants’ attitudes and 
levels of experience. Process variables describe the process or program implementation 
including needs assessment, design, technology, and the types and levels of interaction and 
learner support provided. Program outcome variables include participant satisfaction, 
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performance, learning, and impact. This framework encompasses a broad listing of 
evaluation questions. More complex questions typically involve examining the 
interrelationships between variables from two or more of these categories. 

Online Participation, Interaction Patterns, and Group Bynamics 

A question frequently asked when evaluating online networks is: How can we 
describe online participation, interaction patterns, and group dynamics? In relation to this 
question, we will critically analyze the model developed by Levin, Kim and Riel (1 990) for 
analyzing instructional interactions on electronic message networks and the content analysis 
model developed by Henri (1992). 

Levin, Kim and Riel’s (1990) model has four dimensions of analysis: (a) Participant 
Structures Analysis, (b) Intermessage Reference Analysis, (c) Message Act Analysis, and (d) 
Message Flow analysis. This model was used by Gunawardena, Gittinger, and Dvorak, 
(1991) to evaluate a computer conference for medical students. This team found 
Intermessage Reference Analysis difficult to apply in the specific context; use of the 
technique generated a large message map that was difficult to decipher. In the experience of 
the present authors, even when the interrelationships among messages can be mapped out 
clearly little is revealed about the learning processes of participants. Levin, Kim and Riel 
(1990) acknowledged that patterns of message interrelationship sometimes had more to do 
with the dynamics and/or power structure of the group than with the cognitive freight of 
messages. For example, messages posted by the teacher generated a disproportionate 
number of responses. Likewise, a second technique presented by Levin, Kim and Riel, 
Message Act Analysis, dealt more with interrelationships among the participants than with 
development of course topics. 

Message Flow Analysis is Levin, Kim and Riel’s (1990) third technique for analyzing 
online interaction among course participants. This technique is based on tallying the number 
of messages exchanged by group members within specific time periods during the 
conference. This technique can be used to produce a graphic representation of the flow of 
participants’ interest in a conference. Typically conferences start slowly, then experience a 
period of intense activity, and then gradually taper ofF to a desultory trickle. Levin, Kim and 
Riel thus provide quantitative verification of the participation patterns familiar by experience 
to most people who regularly participate in computer conferences. This technique seems 
most promising when we are looking for unusual patterns, as for example when we are 
attempting to determine whether a redirection of topic reverses the downward slide of 
interest. When used in this way, Message Flow Analysis may show the effects of message 
content, but the technique does not deal directly with content, per se. We feel it is difficult to 
prove that learning is taking place without looking at the evolution of message content (if 
any) during the progress of a conference. 

Sudweeks and Simoff (1999) discuss an approach similar to Levin, Kim and Riel’s 
(1990) in that they advocate use of multiple techniques and perspectives. Within this 
approach, the techniques they piresent for dealing specifically with CC include "Neural 
Network Analysis." This appears to be an extension of the Intermessage Reference Analysis 
procedure described above, originated by Levin, Kim and Riel (1990) , with the added 
feature that Sudweeks and Simoff apply numerical valuations to the strength of 
interrelationships between messages to produce a dynamic model rather than a map. The 
position of each message in time also forms part of its description in this system, although in 
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our experience the timing of messages in any asynchronous forum tends to reflect the 
demands on participants' time of their lives and work rather than the degree of participants' 
interest in the conference. Levin, Kim and Riel’s (1990) Message Flow Analysis appears to 
provide a somewhat more realistic view of the growth and decline of interest in a 
conference. 

While by adding numerical valuations to intermessage analysis Sudweeks and Simoff 
present a quantitative method of content analysis, they also present a qualitative method of 
analyzing the "dimensions" of a conference. The dimensions they identify overlap to some 
degree the message functions identified by Henri (1992) discussed later. It would be 
interesting to compare the use of the two systems on a single data set. 

Informed by cognitive-information-processing theory, Henri (1992) has developed a 
content analysis model that provides a sophisticated, multi-dimensional analytical framework 
for evaluating online participation, interaction, and the learning process. Citing the “richness 
and efficiency” (Henri, 1992, p. 120) of computer conference transcripts in understanding the 
exteriorized internal cognitive learning process, she proposes examining the content of 
messages as an appropriate analytical method to identify the learning processes and strategies 
selected or developed by learners. Rather than mapping the messages, Henri suggests 
dividing messages into “units of meaning” and classifying these units according to their 
content or function within five dimensions, (a) participation and social interchange, (b) 
interactive content, (c) organizational intent, and application of (d) cognitive or (e) 
metacognitive skills. 

The participative dimension is evaluated quantitatively and qualitatively by 
numbering messages and determining those statements related to the formal content of the 
lesson. Messages not related to formal content of the subject matter are coded as social. 
These statements are critical to the building of community, establishing social presence, and 
group cohesion. Henri’s model then distinguishes between interactive versus non-interactive 
and explicit versus implicit interaction. This chain of interaction is represented as a three 
step process whereby 1) interlocutor A speaks to B; 2) B responds to A; followed by 3) A’s 
answer to B. Organizational functions are statements that organize the learners in the 
accomplishment of educational tasks. Cognitive message units exhibit knowledge and skills 
related to the learning process in contrast to metacognitive statements showing self control or 
awareness and that demonstrate strategies relative to the self-regulation of learning. 

In applying this model to the evaluation of an online debate discussed later, a number 
of shortcomings were found (Gunawardena et al, 1995). First, the model appears to be based 
on a teacher-centered instructional paradigm. For example, Henri states (p. 123) that “the 
educator can offer input at three levels: what is said on the subject or theme under discussion, 
how it is said; and the processes and strategies adopted in dealing with it... The educator may 
favor one or another level, according to his or her pedagogical aims and intentions.” The 
paradigm described by Henri is clearly one that is widely applied, as many educators drawn 
to distance learning continue to recreate the traditional patterns of instruction in the new 
medium. 

The second difficulty encountered by the authors in applying the model arose from 
the limited definition and treatment of the concept of interaction. Henri states that messages 
are either “monologic” or “interactive” and suggests further analysis based upon observing 
whose messages gamer the most response. This is somewhat similar to Levin, Kim and 
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Riel’s (1990) “message maps.” However, this type of analysis is merely descriptive of 
individual student performance and the pattern of messages. It offers little insight into the 
contribution individual messages make to the emerging totality of constructed knowledge 
(Gunawardena et al, 1995). The analysis indicated the presence and frequency of 
participants employing cognitive skills, but was unable to explain the learning process taking 
place within the group through the process of interaction between participants. 

In addition, the “unit of meaning” in the model is undefined. It is not based on a 
syntactical unit such as a phrase, sentence or paragraph, rather it is subject to an 
interpretative judgement oh the part of the researcher. This was especially problematic when 
distinguishing cognitive from metacognitve units. The purpose of an easily identifiable unit 
is that multiple coders can reliably and objectively recognize the unit for recording, leading 
to an evaluation model that produces an interrater reliability that increases confidence in the 
validity of the data description. 

The model is perhaps best suited for evaluation within an objective-based learning 
environment of defined membership However, is it also quite clear that such a paradigm is 
inappropriate for evaluating the quality of a learning event designed within a collaborative, 
social constructivist learning environment. 

In concluding this section it is of interest to note a study in which McDonald and 
Gibson (1 998) used transcript analysis to examine interpersonal dynamics and group 
development in computer conferences (CC). Research suggests that the nature of 
communication and group process is changed when interaction is computer-mediated, yet 
there is little evidence in the literature, other than anecdotal, that demonstrates groups can 
and do develop online. Using an adaptation of Henri’s model, this study suggested that it 
was possible to identify specific criteria for use in content analysis with which to study group 
dynamics and development within an online environment. As Brookfield (1990) observes 
group dynamics are an important factor to be considered when planning for discussions and 
group work. This would include such considerations as the degree of cooperation, how the 
group organizes itself and how the dynamics of group cooperation are achieved. Evaluation 
questions for the future will likely include how do online learning networks promote 
collaborative learning? How does group development impact peer exchange and the social 
construction of knowledge? 

Learner Satisfaction with the Online Learning Experience 

Evaluators are often interested in determining whether learners were satisfied with 
their online learning experience. Did they feel that it had been a worthwhile use of their time, 
and one that they would be willing to repeat? We will discuss two studies (Gunawardena and 
Zittle, 1997; Gunawardena and Duphome, 2000), that used survey data and regression 
models to answer the question: What are the variables that can predict learner satisfaction in 
online learning networks? Both studies are based on survey data from an inter-university CC 
that provided a forum for graduate students in distance education to share and discuss 
research, and experience distance education by using computer mediated communication 
(CMC). Gunawardena and Zittle (1 997) used eight process variables to predict overall 
student satisfaction with the CC. Gunawardena and Duphome (2000) explored predictors of 
learner satisfaction in a CC basing their analysis on the Adult Distance Study Through 
Computer Conferencing (ADSCC) model developed by Eastmond (1994). In this model, 
Eastmond identified three major aspects which sequentially influence the student’s study 
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experience: (a) readiness -- the personal and environmental factors that prepare the student 
for study in this instructional situation; (b) online features — the unique elements that make 
up the computer conferencing environment; and (c) learning approaches — the general and 
specific learning strategies a student uses to make the conference an effective learning 
experience. Gunawardena and Duphome tested the predictive power of the Eastmond 
ADSCC model by examining which of the three independent variables: learner readiness; 
online features; and CMC-related learning approaches, could best predict the dependent 
variable, learner satisfaction with an academic CC. 

As with any survey-based research, both studies relied on learner self-reports in 
evaluating learner satisfaction. This always carries some potential for error; either learners 
may be tempted to answer in the way they feel is somehow more creditable or they may find 
that the way in which survey questions are asked does not fit their experience well. The use 
of qualitative data-in this case, allowing the participant to describe the conferencing 
experience in his or her own words - is a good counterbalance. 



Knowledge CoHistractnoe 

Perhaps the most challenging question one can ask in evaluating online learning is: 
Was knowledge constructed within the group by means of the exchanges among 
participants? We will discuss two interaction analysis models we have used to answer this 
question. Initially, we began to use Henri’s (1992) model to analyze the transcripts of a 
global online debate we had conducted as an adult professional development experience. 
However, it became clear that three aspects of Henri's model; its basis in a teacher-centered 
instructional paradigm, its distinction between the cognitive and the metacognitive 
dimensions, and its treatment of the concept of interaction, were unsuited for application to 
the debate. 

In response to the perceived limitations of the Henri model, Gunawardena, Lowe, and 
Anderson proposed anew model (1997) for use in evaluating the construction of knowledge 
through social negotiation in online learning networks. This model developed a new 
definition of interaction for the online learning environment and used the metaphor of a 
patchwork quilt to better describe the process of shared construction of knowledge that 
occurs in a constructivist learning environment. Based on this new definition of interaction, 
and grounded theory principles, an interaction analysis model was developed that included 
five phases, reflecting the process of negotiation which must occur in a social constructivist 
learning environment. The phases of learning outlined in this model occur at both the 
individual and social level and can be described as: Phase I: Sharing/Comparing, Phase IT. 
Dissonance, Phase ni: Negotiation/Co-construction, Phase IV: Testing Tentative 
Constructions, and Phase V. Statement/ Application of Newly-Constructed Knowledge. 

Two major themes were observed related to knowledge construction when applying 
the model to the analysis of a debate. One was the progress of certain strands of argument 
from Phase I to Phase V, which can be described as an exercise in the co-construction of 
knowledge, moving from lower to higher mental functions. The other was the evidence of 
more than one and sometimes three phases within a single message posted by one participant, 
which usually progressed in sequence through the phases, showing progress from lower to 
higher mental functions, showing how individuals contributed toward the co-construction. 



As the model has been applied to a succession of conferences, however, it has 
brought to light indications that many, perhaps most, conferences do not proceed beyond 
Phase n. Very often, participants share and compare their experiences and viewpoints on the 
course topic, and then when they find areas of disagreement they appear to “agree to 
disagree,” rather than proceeding to negotiate new meaning as the social constructivist theory 
would suggest. It is still unclear how this discovery may be used to improve techniques for 
moderating conferences, or making whatever other changes are needed to increase the 
likelihood of new knowledge construction. Other drawbacks which may be encountered in 
using this method include the fact that it is time consuming and the fact that some message 
elements may be difficult to assign with certainty to a specific phase thus lowering interrater 
reliability. 

Issues to Comttemi Analysis 

Analysis of the actual content of messages submitted and archived within an online 
learning network experience is a logical, appropriate and recommended means of evaluating 
the quality of the learning experience (Mason, 1991). Content analysis is undoubtedly one of 
the most powerful and important research techniques in the social sciences (Krippendorf, 
1980) and holds high promise as an analytic method for the evaluation of the quality of 
learning in an online learning network. It is also a method found to be methodologically 
challenging to evaluators. 

Content analyses are comprised of four basic steps. The first is a compilation or 
selection of the transcript(s) for analysis. Unique to the online learning network is the 
automatic creation of text-based archives of communication interactions that constitute the 
online forum. Downloading or accessing these files provides a ready data source for the 
evaluation. Second, the researcher develops a coding scheme for empirically identifying 
instantiations of the variable of interest and trains coders to implement the protocol. 

Following the transcript coding, coding decisions are compared for interrater reliability, that 
is, the extent to which different coders, coding the same transcript, reach the same coding 
decision. The final step is an analysis and reporting of the coded data either in describing the 
variable(s) or identifying relationships between variables. The extent to which the analysis 
and findings are credible and accessible to the evaluation audience is directly related to issues 
of reliability and validity. According to Potter-Levine Donnerstein (1999), “If content 
analysts cannot demonstrate strong validity and reliability for their findings, then people who 
want to apply these findings should be wary of developing implementations, especially when 
resource costs and client expectations are high. ”(p. 258). 

Reliability 

Establishing reliability begins with the development of a coding scheme likely to 
activate similar psychological schema within different coders and result in a high degree of 
similar decisions among these coders as they assign message units to categories determined 
by the researcher as an instantiation of the construct of interest. Following coding, 
calculations are made to determine the level of agreement in coding decisions across coders, 
i.e. interrater reliability. Three difficulties are apparent. First, many studies simply do not 
report interrater reliability statistics leaving reliability in question. Second, while higher 
levels of agreement are obviously desirable, the exact level of interrater reliability that must 
be achieved has not been clearly established, forcing researchers to decide acceptable levels 
for themselves and reporting results largely uninterpretable for evaluation stakeholders. 
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Further, choosing a method of calculation is problematic. The most common method of 
reporting interrater reliability is the calculation of a simple percent agreement statistic 
(Holsti, 1969), yet a number of methodologists declare this practice inadequate by failing to 
account for chance agreement (Krippendorf, 1980) recommending far more conservative 
statistics; i.e.Cohen’s kappa (k) or Krippendorf s alpha. Still others debate these statistics as 
too conservative and constraining to the researcher. See Potter and Levine-Donnerstein 
(1999) for a detailed explanation. 

Validity 

Further exacerbating this dilemma is the issue of the trade-off between reliability and 
validity. As Krippendorf notes, reliability often gets in the way of validity (1980). The 
design of a coding scheme sufficiently detailed as to provoke few disagreements across 
coders in assigning coding decisions often lowers ecological validity. Establishing validity 
in content analysis is a two-step process: first, developing a coding scheme that is faithful to 
the focal concepts of the theory and second, assessing the decisions made by coders against 
some standard. According to Potter & Levine-Donnerstein (1999), if codes match the 
standard for correct decision-making, then the coding is regarded as producing valid data 
Yet, surprisingly little attention has been given to the validation issues surrounding the 
development of coding schemes. Determining validity implies the existence of some sort of 
standard or sound theoretical basis for the evaluation of social learning that occurs within an 
online learning network. Without a descriptive model of the ideal collaborative learning 
process we cannot even begin to identify empirical indicators or instances that will form the 
basis of a coding scheme as a standard against which to evaluate whether or not effective 
learning is occurring within the network. 

Potter and Levine-Donnerstein propose the resolution of these issues based upon 
carefully developed coding schemes that sufficiently activate focal concepts of theory and 
provoke a high degree of congruent judgements (i.e. intersubjective congruence) thus 
maintaining both ecological validity and high reliability. In surveying the methods and 
models employed in the evaluation of online learning networks, the exploitation of content 
analysis techniques in evaluation studies is particularly inviting. In our current studies we 
are attempting to develop coding tools that are reliable, efficient, effective, and valid in the 
empirical investigation of online learning. 

The Systems Component Contexture ModeD 

Thus far this paper has presented an overview of the approaches used and challenges 
faced by the authors in the evaluation of online learning networks and focused on answering 
three frequently asked evaluation questions. Each of these questions provide the evaluator an 
opportunity to address a number of variables: descriptive, process, and outcome albeit in a 
decontextualized manner. Indeed, much of the evaluation and research in online learning is 
based upon focused inquiry directed at one or another variable comprising only one or two 
components of the network often disregarding the organizational context in which it is 
embedded. This has led to a somewhat fragmented understanding of the functioning of the 
socio -technical system as a whole. 

The very complexity of the interrelationships of these variables has led us to develop 
a model based upon systems theory and extant research that can be used as a unifying 
conceptual tool for component analysis and further research. As Coldeway (1988) has 
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suggested, component analysis is an especially effective way to combine the benefits of 
evaluation with a potential of more generalizable research. Component research formatively 
analyzes each component in a course or a system to determine the overall impact or level of 
contribution each makes to the larger picture of the learning network. Evaluation questions 
are formulated to answer questions within or across components. Adding the multi-level 
analysis to the component evaluation model, represents an attempt to make explicit the 
cultural and organizational setting in which the technology of the online learning network is 
employed. This provides the foundation for consideration of the effectiveness of the online 
learning events at a number of levels simultaneously, and the identification of the trade-offs 
between them. To what degree, for example, does the online learning network support the 
social learning desired and to what extent does this paradigm complement existing 
institutionally supported pedagogical practice? To what extent is it economically feasible to 
train instructors and design teams to adapt to new collaborative learning philosophies in 
order to utilize this virtual medium to its fullest potential? 

The System Component Contexture Model provides the opportunity to integrate 
various knowledge bases in addressing not only these questions but also other equally 
important variables, by incorporating theoretical notions such as interaction, social presence, 
transactional distance and learner control. It consists of examining the level of influence of 
four components of the online learning network at three levels. By explicitly distinguishing 
the components at different levels, the interests and impact of all participants and variables 
within the context are brought sharply into focus. In our current work, we distinguish 
between and highlight the importance of researching all component processes that contribute 
to and impact an online learning environment - including teaching, learning, communication, 
design, management, institutional mission, and socio-cultural context. Few studies document 
these components in relationship to the effectiveness of the overall program of which they 
are a part. Our model incorporates the components of the system; the institutional context, 
and the instructional experience, connected by extensible bands within a resiliency 
framework. 

The function of the evaluation then may be extended to include a systematic 
monitoring of the pressure pulls of each component and the ultimate effect on the learner. 

The model accounts for the dynamic connection of each component at multiple points; 
represented by ovals in the diagram (see page 12.) These connections exist in a state of 
pressure. In a robust system, a state of equilibrium (symmetry) exists. At any given pressure 
point, however, an ineffective, inefficient, or inappropriate practice, method, or preparation 
elicits a pressure pull on the extensible band (connection) connecting one component to the 
system. While these pressure pulls may originate from any level of any component, their 
interconnectivity results in an effect that is ultimately felt throughout the system and is 
therefore, detectable at several levels. Measurement is not always systematic, hence the need 
for frequent, formative, and multiple methods of measurement. An example may serve to 
enlighten these dynamics; the voice of student dissatisfaction with delivery technology, i.e. 
synchronous versus asynchronous software (Level 3/ Context Component decision) may be 
clearly heard in a survey-based evaluation of an online course (Level 1/ Control Component 
impact). 

Each component possesses a certain inherent ability to adjust to ineffective elements 
within it. This ability is component resiliency. Just as an extensible band can accommodate 
and adjust to changing pressures, so too can a component adjust and maintain the integrity of 
the system. Ultimately, if the pressure exceeds the ability of the band to accommodate the 
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pressure (resiliency), one or more of the bands will disconnect at the defining point. In an 
online learning network as the upper level of the resiliency factor is approached, the 
component is rendered increasingly ineffective. Due to the interweaving of the conjoined 
bands an negative impact will be exerted on the entire system equilibrium. 

Levels 

The first level of influence is institutional experience, which encompasses the 
educational philosophy, mission, resources, and management of the institution within which 
the online learning network program is housed. The next level incorporates all aspects of the 
mediated curriculum including the epistemology of the instructors, the course design team, 
the knowledge resources, the choice of media, and the activities designed in support of the 
learning events. The final level is the learner level that includes, learner characteristics, life 
situation, prerequisite skills, attitudes, and goal orientation. 

Components 

The four components of the model are: Content, Context, Collaboration, and Control. 
The content component is concerned with the topic area of the course or program being 
evaluated. Evaluation at this level concerns: the place of this course or program within the 
institutional mission (mission-specified program, core program, elective course, pilot project, 
course component, conference etc.) and the relationship of the program to the achievement of 
the institutional goals. From the instructional level, the questions center on course structure, 
instructional design, instructional goals, objectives, and anticipated outcomes relative to the 
specific subject matter, as well as sources of knowledge and available resources. On the 
learner level, evaluators address prerequisite skills, attitudinal factors specific to content, and 
goal orientation. 

The context component examines situational, environmental factors. At the 
institutional level, evaluation questions address the management of the institution, cost- 
benefit analysis, support of the delivery technology, support for student, staff, and faculty 
needs, faculty-student ratios, and the ability for increased capacity (i.e. the degree of 
flexibility for program expansipn). Second level questions look at professional development 
for staff and faculty, faculty experience with technology, and the choices of technology 
available. From the learner level, such things as learner support, and technology training are 
considered. 

The collaboration component includes intra-institutional collaboration as well as 
inter-institutional collaboration, including external alliances with like institutions and support 
for cross-departmental course design teams. These factors largely impact instructor-student 
interaction and student-student interaction at the learning event level. The pressure pulls in 
this component are straightforward as evaluators address situational opportunities provided 
for group processes resulting in the social negotiation of meaning. 

The final component deals with issues surrounding learner control. The focus of the 
evaluation of this component is on learning and the assessment of learning. At an 
institutional level, factors will include the assessment philosophy, indicators for student 
achievement and success, entry and exit and external accreditation requirements. At the 
instructor level, evaluation questions will be asked based upon instructor epistemology, 
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learning theory perspective, and choice of instructional design and media. At the level of the 
learning event, learner control, and level of learning will be addressed. 

The model forms the basis for an evaluation plan that provides a contextual 
framework firmly grounded in past distance education research from which to conceptualize, 
analyze, and make decisions related to the teaching/leaming situation within which an online 
learning network is embedded. Descriptive, process, and outcome variables can be examined 
within a social context. This conceptualization can encourage theoretical speculation and 
formulation by reducing complexity as well as providing a means to operationalize relatively 
abstract variables. 

To summarize, the function of a systems-component contexture evaluation is to 
systematically monitor the pressure pulls of each component and the relationship each level 
of each component ultimately has upon the learner. The questions that we ask as we evaluate, 
and the instruments we choose to employ, enlighten and inform us about some aspects of 
each component while leaving us ignorant of other components. The use of this holistic 
model provides a framework from which to determine those questions left unasked as well as 
the opportunity to draft new questions for the future. 



Concision 

In order to improve our understanding of how to use online environments to foster 
learning, it is important to examine the online learning experience from multiple points of 
view. This paper has touched on a number of models which have been suggested by those 
interested in evaluating and researching online learning environments. We believe that no 
single technique for analyzing the quality of online learning experiences can yield complete 
answers. The complex nature of online learning calls for the use of multiple methods and 
multiple sources of data to understand group as well as individual learning. The Internet is an 
ever-changing phenomenon. We advocate the use of multiple approaches in studying online 
learning networks in order to obtain a wider picture of this dynamic learning environment. 
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